status: fix FSMonitor history and clean-proof gaps - #74
Open
ttaylorr-oai wants to merge 29 commits into
Open
Conversation
ttaylorr-oai
marked this pull request as draft
August 26, 2026 18:53
ttaylorr-oai
marked this pull request as ready for review
August 26, 2026 19:17
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 04:33
f5f8eaf to
0a572e5
Compare
with_lock__wait_for_cookie() gives a filesystem provider one second to report a synchronization cookie. A healthy FSEvents stream can miss that deadline while macOS is under load. The daemon then returns a trivial response, and status scans the entire index even though event delivery is still making progress. 4b1c56a (fsmonitor: flush pending FSEvents before cookie wait, 2026-07-21) requested an asynchronous flush on every Darwin query but kept the same one-second deadline. f439708 (Revert "fsmonitor: flush pending FSEvents before cookie wait", 2026-08-17) reverted it after a matched 48-query test still saw 12 timeouts in each arm. Avoid restoring that unqualified hot-path request. When the initial Darwin wait expires, request an asynchronous FSEvents flush and wait one more bounded interval. Successful queries retain the original wait and do not issue a flush or extend their deadline. The asynchronous call cannot block on the callback while the client holds main_lock. If the provider stays silent, retain the existing trivial-response fallback after the retry. Add a test-only callback delay to exercise both outcomes: a 1.2-second delay is recovered, while a 2.5-second delay still reaches the bounded fallback.
The daemon currently assumes that each client which advances an FSMonitor token also updates the repository's canonical index. That does not hold for commands using GIT_INDEX_FILE. A private index can advance the daemon past the canonical index's token and cause the canonical index's next query to receive a global invalidation. Keep a deduplicated overflow batch instead of discarding old paths. Clients at the overflow sequence still get an exact delta. Older clients get a conservative union of paths, which may overreport but cannot miss a change. All paths are interned. Keep a pointer-identity hash set with the overflow batch so later compactions hash only newly retired paths, rather than rebuilding a set over the daemon's lifetime history. Add a regression which advances a private index repeatedly, verifies that compaction remains deduplicated, and then checks that a read-only canonical status reports both changed files without a trivial response.
Retired batches are collapsed into a path-only overflow set. That keeps old indexes complete, but it loses the sequence in which each path was last observed. A client that consumed an inode event can therefore see it again after another index compacts the batch list, causing repeated hard-link scans. Unpinned batches have a zero pinned time and are also eligible for compaction immediately despite the default grace period. Do not use unpinned batches as truncation boundaries. Record the newest original batch sequence for every overflow path, and filter overflow responses against the client's requested sequence. The normal batch walk remains unchanged; sequence lookups are confined to overflow responses. Cover both the default retention grace and the cross-index hard-link case. The latter persists a nonzero checkpoint, compacts through a private index, and verifies repeated canonical reads do not rescan or fall back to global invalidation.
The delayed-cookie tests send the v1 timestamp token "0" and only check that the response is nonempty. Both recovery and fallback can satisfy that assertion with the same trivial response, so the tests do not distinguish a rescued cookie from a token-generation reset. Send a deterministic valid v2 token instead. Verify that the 1200ms case preserves its token generation without a global invalidation, while the 2500ms case changes generation and sends the fallback invalidation.
215845a (fsmonitor: preserve authenticated proofs across ordinary commands, 2026-08-15) enabled the clean-status history handoff for merges, but excluded invocations where fast_forward was FF_NO. Requested merge topology does not determine whether the resulting index is semantically safe. A clean non-fast-forward merge can carry the same authenticated FSUC/FSCF state as a fast-forward merge. As a result, --no-ff, --no-ff --no-commit, and merge.ff=false all dropped FSUC and reduced the FSCF flags from 15 to 9 after a clean merge. Each subsequent read-only status invalidated the external history and rescanned the semantic manifest. Enable the handoff for every merge using the canonical index. Conflict handling still invalidates unsafe proofs, and explicit alternate indexes remain excluded. Cover all three non-fast-forward forms, repeated read-only status calls, conflicts, and alternate indexes.
An exact clean status can repair a stale FSMonitor checkpoint or cached stat data while it scans. The repair requires an index write, so the existing issue path leaves no clean sidecar behind. Read-only callers then repeat the full scan until a second writable exact status publishes the proof. After the repair is written and resumable history is durable, install a sidecar bound to the rewritten index. Keep optional-lock-disabled commands read-only, preserve the literal exact-command restriction, and do not extend sidecar support to linked worktrees. Cover repeated read-only scans after a legacy daemon replacement, the single writable index repair in main and linked worktrees, and the next read-only sidecar hit in the main worktree. Keep option-bearing status commands ineligible for proof publication.
A configured pull can discard each layer of authenticated status history even when worktree inputs remain unchanged. Command-scoped protocol and HTTP settings change the config digest, directory events with more than 64 tracked descendants reject the semantic proof, and a fast-forward which adds an indexed directory drops the paired untracked cache. The next status can consequently preload and refresh the full index. Treat command-scoped protocol and HTTP settings as transport-only. For a large directory event, authenticate each distinct attribute source once instead of rejecting the cone outright. When a checkout adds tracked paths, retain the paired untracked cache and replay those additions through its existing invalidation path. Cover configured pulls in main and linked worktrees, large directory events, nested attribute-source changes, and branch switches which add tracked directories. The conservative full-scan fallback remains in place when an attribute source changes.
Configured pulls preserve FSMonitor clean proofs when checkout can authenticate every index change. Tracked policy files were an exception: adding or replacing .gitattributes or .gitignore made the generic semantic transfer reject the whole proof. Later read-only status commands then had to rescan the worktree and could not restore the paired untracked proof. Let checkout retain history across regular policy-file changes that it writes itself. Attribute changes refresh the worktree manifest before the provider boundary is rebound, and fail closed if that refresh cannot authenticate the new sources. Keep the existing untracked-cache invalidation for ignore changes, and transfer that cache only while the full tracked proof remains current. Exercise configured fast-forward pulls in main and linked worktrees. A required-filter control also verifies that changed attributes invalidate the affected tracked entry instead of certifying it.
A clean status proof can survive a pull only when its configuration, tracked-file state, FSMonitor token, and paired untracked cache still describe the resulting worktree. Command-scoped push transport settings were included in the configuration fingerprint. Checkout could also discard the untracked proof for policy-file changes or leave events from its own worktree writes outside the proof. The next diff, write-tree, or status then repeated tracked and untracked work. With optional locks disabled, status could not publish the repair, so each invocation paid the same cost. Treat push.negotiate and remote.*.pushurl like other command-scoped transport settings. Preserve the paired untracked cache across checkout, invalidate only affected policy scopes, and authenticate distinct attribute-source directories before transferring semantic history. For checkout, reset, merge, and sequencer worktree updates, write a provisional index under the existing lock, consume the daemon events caused by the update, and certify the result against that locked index before the final write. This also covers stash cleanup through its hard reset. Alternate indexes, split or sparse indexes, unsafe filter or manifest state, and incomplete stat data still fall back. Cover configured pulls, root and nested policy changes, main and linked worktrees, rebase, reset, stash, checkout, and repeated read-only status.
The Linux listener queued every inotify event as a file pathname. Directory events therefore lacked the trailing slash used by semantic invalidation. After an owned worktree update retained a clean proof, a following read-only status could not close those events against it. The command scanned all tracked entries. With core.preloadIndexBulk enabled, this work appears as statx calls instead of lstat counters. Format Linux worktree events through fsmonitor_format_worktree_paths() and use IN_ISDIR to preserve their directory identity. Advertise directory metadata support and mark Linux tokens so clients replace daemons using the old event format. Concurrent clients can see an expected connection reset while one client replaces a stale daemon. Silence that diagnostic only for gentle IPC reads, which already reconnect, without changing ordinary IPC error handling. Cover both bulk preload modes plus single and concurrent daemon replacement.
Bulk index preload can defer content and conversion checks to the diff that normally follows refresh_index(). repair_fsmonitor_proof() only refreshes the index before deciding whether to persist a clean proof; it does not run that diff. With core.preloadIndexBulk enabled, a pull or rebase that changes .gitattributes or .gitignore can therefore leave tracked entries dirty after the writer reports a successful repair. Repeated read-only status calls cannot persist the missing repairs. Do not request deferred bulk results in the writer-repair path. This keeps the ordinary status and diff bulk path unchanged while forcing the exceptional repair to finish its tracked checks before certifying and writing the proof. Enable bulk preload in the existing fast-forward, policy-file, and sequencer writer tests. They verify targeted refreshes and two subsequent read-only status calls without scans or index writes.
A writable Git command can leave a complete FSMonitor proof in a repairable state when it changes policy files, adds or removes an intent-to-add entry, or delegates the final index write to a child process. The existing repair path assumed that the manifest and untracked cache remained closed. The sequencer also kept its stale in-memory index after git commit rewrote the canonical index. Stash operations and completed rebases could therefore drop FSUC or overwrite the child's newer token. Read-only status could not persist the repair and repeated tracked or directory work. Let index-only writers refresh changed manifests and rebuild the paired untracked cache against the provisional locked index. Preserve unrelated history for safe intent-to-add changes and unmerged non-attribute paths, then reload the canonical index after child writers before repairing it. Active filters and unresolved structural indexes still fall back. Linux can report an event for a watched directory without a child name. Keep the watched directory in that case, encode its token capabilities in the order understood by Linux clients, and serialize incompatible daemon replacement on Linux as on macOS. Cover stash creation and application, policy-file updates, ordinary and --rebase-merges conflict completion, cherry-pick's deliberately weaker tracked-only proof, nameless inotify events, and primary and linked worktrees. Repeated optional-lock-free status calls must not rewrite the index or rescan tracked entries.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 21:04
0710046 to
1ecb949
Compare
snapshot_open() initializes its output before passing the path to open_nofollow(). The manifest builder can ask it to pin a synthetic repository while rejecting a sparse index. Such a repository need not have an index path. Ordinary Linux happened to return EFAULT from open(NULL), but passing NULL violates the contract of open(2) and aborts under UBSan. Reject NULL and empty paths after initializing the snapshot. The existing clean-status-manifest sparse-index test exercises the fail-closed result.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 22:24
1ecb949 to
08f9074
Compare
766fce6 (simple-ipc: split async server initialization and running, 2024-10-08) separated server initialization from startup so owners could finish setup before accepting clients. But start and stop still inspect lifecycle flags without shared synchronization. A late start can therefore release workers after cleanup requested shutdown, and concurrent cleanup paths can repeat the stop sequence. Two transport races compound that problem during daemon replacement. An interrupted shutdown wake can publish the shutdown transition without waking accept(), leaving the stop path hung. A client whose accepted socket is closed before its request write can die from SIGPIPE before the gentle EPIPE recovery runs. Serialize the first start and stop, queue the complete shutdown wake before publishing that transition, and contain SIGPIPE within gentle client writes while preserving the caller's signal state. Exercise late startup, repeated stop, interrupted wakeups, and concurrent writes to a peer that closes immediately after accept().
The Darwin daemon treats delivery of its cookie-file event as proof that all earlier worktree changes have been published. That assumes the callback containing the cookie cannot overtake logically older work. A retained FSEvents trace disproves that assumption. The cookie callback completed before a later callback published removals that had happened before the cookie was created. A status query could therefore answer from incomplete event history and report a dirty worktree as clean. After the ordinary cookie wait, ask a long-lived worker to flush the FSEvents stream and then drain its serial callback queue. The flush schedules provider events; the queue drain waits for those callbacks to finish publishing. Accept the boundary only when the cookie was seen in the same token generation, and coalesce overlapping requests onto a single fence. If the bounded fence times out or intersects shutdown, return a conservative result and retire the daemon before unsafe stream teardown. Advertise the stronger boundary as a capability and token suffix so new clients replace unfenced daemons while older clients retain prefix compatibility. Exercise split and blocked callbacks, timeout replacement, generation reset, listener shutdown, concurrent coalescing, second-wave requests, rename and cache scopes, and protocol compatibility. Keep status proof tests outside the split-index matrix where that proof is deliberately disabled, and materialize externally restored tokens before raw-index helpers consume them. The provider fence adds work to each Darwin query, while overlapping queries share a fence when their cookies are already registered.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 22:40
08f9074 to
7bed8a3
Compare
ae161e8 (fsmonitor: validate builtin daemon responses before applying them, 2026-07-10) validates each worktree path with verify_path(). That helper enforces index-entry rules and rejects a .git component anywhere in a path. Filesystem providers can legitimately report such a component for an untracked nested repository. The client therefore rejects the entire response after an event such as scratch/.git/file, forcing a full worktree scan. Commands that need a current provider boundary cannot persist a clean proof from that query. Validate the narrower daemon-response contract instead: require a relative path with nonempty, non-dot components and at most one trailing separator. Keep rejecting absolute and traversal paths, but allow .git components that already exist in the worktree. Cover the parser directly and exercise a real Linux daemon event against an optional-lock-free status oracle.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
2 times, most recently
from
August 28, 2026 05:07
ad39dc8 to
2ebd9f8
Compare
fbd7a23 (rebase: introduce and use pseudo-ref REBASE_HEAD, 2018-02-11) records the commit currently being replayed. The sequencer normally deletes that ref before executing each todo item. When the final item stops for a conflict, rebase --continue commits the resolved result. pick_commits() then reaches the end of the list without entering another iteration and removes the rebase state directly. The merge backend skips finish_rebase() because the sequencer owns cleanup, so REBASE_HEAD survives a successful rebase. The same path also strands the fsmonitor proof when index.skipHash is enabled. After committing the resolution, the sequencer reloads the canonical index and repairs its proof through a close-only index.lock witness. A skip-hash witness has a null trailer and a fresh file identity, so proof-epoch validation cannot bind it to the in-memory index. Rebase succeeds without FSUC, and read-only status cannot repair it. Delete REBASE_HEAD whenever interactive-rebase state is removed. This matches finish_rebase() cleanup and also avoids retaining a ref for an explicitly quit operation. Propagate a failed deletion so rebase does not report success after leaving the stale ref behind. Give only PROVISIONAL_LOCK witnesses a real checksum. The final index rewrite continues to honor index.skipHash, preserving the normal index write fast path while giving proof repair an authenticated epoch. Extend the final-conflict test to require REBASE_HEAD during resolution, its removal after completion, and a reported failure when the ref cannot be deleted. Exercise skip-hash proof repair after a clean-prefix, conflicted replay in primary and linked worktrees, with plain and configured-filter repositories.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 28, 2026 05:34
2ebd9f8 to
16d98d8
Compare
A proof repair can close one provider token, collect untracked results,
reopen the token, and close it again with the same struct wt_status. If
the first closure published untracked output, the second closure tries
to publish another snapshot over it and hits:
BUG: publishing untracked results over collected status
This is reachable from stash pop when an index writer repairs a complete
FSMonitor proof while an untracked path is present.
Before closing a required new token, discard output explicitly marked as
coming from an earlier authenticated token closure or bulk preload.
Keep the BUG for ordinary caller-collected results, which must not be
silently overwritten. Extend refresh invalidation to discard both
authenticated forms as well.
Allow the scripted provider to opt into proof repair, and add a
regression covering the two-token stash path with visible untracked
output.
An fsmonitor provider can reset while merge is reading an index with an authenticated clean-status proof. The reset leaves that proof available for revalidation, but merge updates the worktree before repairing it. The resulting index can lose FSUC after a clean merge. Read-only status cannot persist the missing proof, so every later status falls back. A multi-strategy merge can lose the same history after preparation. An external strategy may replace the index before it declines or reports a conflict. restore_state() then reloads that index while rewinding the worktree. A later built-in strategy sees the original repair decision, but no longer has the paired proof from which to repair. Resolved conflicts expose a separate instance of the same failure. merge clears the resolve-undo extension before updating the worktree. That removal sets RESOLVE_UNDO_CHANGED, which prevented checkout from transferring an otherwise current proof. The result had neither a live provider token nor a pending token from which the writer could repair. Revalidate an authenticated proof before a non-fast-forward merge updates the worktree. Repair it before built-in results are published, after successful external strategies, and after each restore_state() rewind. Permit transfer after the resolve-undo map has been cleared, since removing that optional extension changes neither tracked entries nor worktree contents. Continue rejecting a live resolve-undo map. A repaired writer stats only entries that lack provider validation or stat data before certifying the new index. Fast-forward merges retain their existing path, while conflicts continue to fail closed. Cover built-in ort with and without retained resolve-undo history, trivial and content-level resolve merges, an external strategy that declines, and one that leaves a three-stage conflict before a clean ort retry. Verify that clean results remove resolve-undo data, keep a paired proof, and leave repeated read-only status unable to rewrite the index.
378744b (status: reuse closed proofs for scoped queries, 2026-08-11) taught the untracked cache to reconcile a provider-reported direct child without reopening its directory. A valid cached directory can still have a null exclude_oid when its existing contents are all tracked, since traversal never needed to load its tracked .gitignore. prep_exclude() interprets that null OID as proof that no per-directory exclude exists. The targeted refresh can therefore report a newly created ignored file as untracked. Before refreshing a provider-dirty cached directory with a null exclude OID, use its tracked exclude as the expected identity and load the worktree source. Prefer the exact stage-zero entry, then look for a case-folded alias on case-insensitive worktrees. add_patterns() still opens and hashes the actual source when only an alias exists, so a case collision can only force invalidation. Use the empty-blob ID for an unmerged, non-regular, removed, or intent-to-add match so that it forces a source read and conservative invalidation unless the source is truly empty. Cover both exact and case-folded tracked excludes with read-only status calls. They must match a cold oracle without opening the directory or writing the index, and a changed source must invalidate the cache. Also pin the sparse-index boundaries: an in-cone event retains targeted refresh without expansion, while a vivified outside-cone source takes the existing conservative expansion path.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 00:24
fd910c1 to
b4d251b
Compare
The provider fence added in 7bed8a3 (fsmonitor: fence Darwin callbacks before answering queries, 2026-08-27) calls FSEventStreamFlushSync() from a long-lived worker. The client gives that worker one second before it retires the daemon. Under sustained status traffic, the synchronous provider call can cross that deadline and return immediately afterward. The timeout still forces a daemon restart, and the next status conservatively scans the worktree. A retained trace showed this turning a clean status into a 17-second outlier. On local APFS and HFS volumes, register sticky vnode watches on each watched root and its canonical ancestors before starting the FSEvents stream. Use FSEventStreamFlushAsync() and wait until the callback has published through its returned event ID and a serial queue barrier. The worker waits on its existing condition variable, so the bounded timeout can interrupt it without racing an uncancellable provider call. WatchRoot notifications have event ID zero and cannot be represented by that monotonic token. Treat the kqueue poll as the fence's linearization point, and reject the fence if any watched namespace edge occurred or a watched root changed identity. This also covers a root or ancestor moving away and back before the fence completes. Fall back to the synchronous provider fence when the vnode proof cannot be installed, preserving the existing conservative behavior on unsupported filesystems and resource failures. Exercise the positive event-ID wait, the zero-ID rename ABA, the synchronous fallback, and 512 consecutive read-only status calls. Require every stress-test request to complete without restarting the daemon.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 00:48
b4d251b to
837f89d
Compare
An authenticated clean-status sidecar is bound to the identity of the index file it certifies. A stash push can restore a complete FSMonitor and untracked-cache proof after its child processes rewrite the index, but the existing sidecar still names the old inode. The next read-only status rejects it and takes the slower history path even though stash left the worktree clean. Remember whether stash started with a regular, singly linked sidecar. When optional locks are available and no post-index-change hook is configured, retain the status data gathered by proof repair, commit and reread the final index, then issue a replacement sidecar from that same certifying scan. This avoids a second worktree traversal while binding the proof to the final index identity. Other stash paths keep the existing repair behavior. Cover a scoped push with a subsequent read-only status that must take the clean-proof fast path. Let the existing writer-proof test accept both authenticated sidecar hits and coherent-history reuse, since both are valid read-only fast paths.
An authenticated clean-status sidecar is bound to the identity of the index file it certifies. During an interactive rebase, the child commit run by "rebase --continue" can replace that index. The worktree and index are clean when the replay finishes, but the remaining sidecar still names the old inode. The next read-only status rejects it with a fast-index-mismatch and falls back to the slower history path. Remember whether the rebase started with a regular, singly linked sidecar and persistent FSMonitor proof history. After the replay finishes successfully, reread the final index and use the existing writer-proof repair to publish a replacement sidecar. Only do so when optional locks are available and no post-index-change hook is configured, matching the existing stash guardrails. Move the sidecar-presence check and sidecar-capable repair helper into wt-status so stash and sequencer can share them. Cover a conflicted interactive rebase whose continuation must publish a replacement sidecar and whose next read-only status must hit it.
The clean-status sidecar path accepts only the main worktree. A newly created linked worktree therefore cannot publish a proof after a full clean scan. Later read-only status commands rescan the worktree even though each linked worktree has its own index and sidecar path. Accept a linked worktree only when its per-worktree gitdir remains registered in the common directory and the registered path names the current worktree. Reject an alternate worktree paired with a linked- worktree gitdir; the repository fingerprint continues to bind the resolved worktree, gitdir, common directory, index, and filesystem identities. New worktrees are commonly probed with "git status --short". Let the exact top-level --short and -s forms certify empty output. A fresh worktree index can still be racy, so write and re-read it before saving resumable history and issuing the sidecar. This binds both proofs to the new on-disk index epoch. Cover issuance and optional-lock-free reuse in a registered linked worktree, and verify that an impostor worktree sharing its gitdir falls back.
Stash and rebase may repair and reissue a clean-status sidecar after their primary operation has completed. A lock, read, or proof-repair failure in that optional work currently replaces the successful command result. The command then reports failure even though it has already updated the repository and worktree. The repair path also commits its updated index before rereading it and issuing the sidecar. Another writer can replace the index in that gap. The old clean scan could then be bound to the replacement index. A later status could hide a newly staged change. Treat sidecar repair as best-effort after stash and rebase complete. Retain a descriptor-backed snapshot of the index produced by each postwrite clean scan. Require it to match both the reread index state and the canonical index path before installing the sidecar. A failed repair or intervening write therefore omits the cache and falls back to ordinary status without changing the primary command result. The normal sidecar-hit path remains unchanged. Cover lock contention after successful stash and rebase operations, and replace the index at deterministic postwrite barriers in both repair and status issuance paths.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 11:11
23f8d90 to
d5c2374
Compare
Clean-status sidecars require a durable index identity on local APFS. The history behavior is still valid on other filesystems, but two tests required index.csts after their substantive assertions passed and failed during cleanup on Linux. Hardlink metadata events can also arrive before status refreshes the index. If that refresh creates a racy index, status may conservatively withhold a sidecar until a later scan restores its process-local proof. Requiring immediate reissuance made the test depend on provider timing. Gate sidecar removal on local APFS. For the racy-index case, accept either immediate reissuance or the conservative fallback, but require clean output and recovery to a new proof within three status calls.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
5 times, most recently
from
August 31, 2026 07:30
a43fd03 to
02e1d29
Compare
b65fc91 (status: retain the identity of an index it rewrites, 2026-08-17) lets status keep a race-proof receipt for an index it rewrites. Receipt preparation duplicates the writer descriptor so it can hash the final bytes, and requires the in-memory checksum to match the configured null trailer. Three owned write paths can leave an otherwise valid index without a usable clean proof. A worktree-update repair first writes a checksummed provisional index, then reopens the lockfile write-only before the final skipHash write while the index still records the provisional checksum. Receipt preparation rejects both states, so scoped stash cannot publish a sidecar for the index it installs. Worktree add creates its linked index before the new worktree has a closed FSMonitor provider epoch. The index has FSMonitor and untracked-cache extensions, but lacks the authenticated clean-config proof. Later read-only status processes remain correct, but cannot persist that proof and repeat the full fallback on every invocation. A post-checkout hook can also change worktree-specific configuration. Checking the invoking worktree's settings after the hook can therefore skip priming when the hook enables FSMonitor and the untracked cache only in the new worktree. A clean non-fast-forward merge repairs its authenticated index proof before committing, but leaves the existing sidecar bound to the old index and HEAD tree. The next read-only status rejects it with a fast-index-mismatch and scans the semantic manifest. Only a later writable status can replace the stale sidecar. On Apple, add a read-write reopen operation only for the provisional index lock so receipt preparation can read the final index. Fall back to the original write-only reopen when read access is unavailable, allowing the write to succeed without a receipt. Other platforms retain the write-only reopen. Finish provisional writes through a cold helper that restores the null object ID before the receipt-aware write. After worktree add successfully runs the post-checkout hook, read the linked worktree's effective FSMonitor and untracked-cache settings. If both features are enabled, run one silent status to establish the provider epoch and persist the complete proof. Do so only when the caller permits optional locks; never override an explicit --no-optional-locks request. Factor the best-effort sidecar reissue used by rebase into wt-status. After a successful merge commit has installed its final HEAD and index, use that helper to authenticate the settled state. Sidecar failure still falls back to ordinary status and never changes the merge result. Cover receipt publication and adoption after scoped stash, preserve the generic write-only tempfile contract, and require worktree add to honor post-checkout index writes, linked-worktree configuration, and disabled optional locks. Also require a clean non-fast-forward merge to publish a sidecar that its next read-only status can consume.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 31, 2026 08:59
02e1d29 to
8055b6c
Compare
A command can disable worktree-to-Git filters with empty clean and process commands and required=false. It does not need to disable smudge, which converts in the other direction. The status fingerprint only normalizes the four-setting form, so an otherwise equivalent three-part override discards scoped FSMonitor history and forces a tracked-file scan. Recognize the complete read-side override as well. Continue to fingerprint partial or mixed-driver overrides, and retain the normalized-filter bit so that temporarily disabling filters cannot publish a clean sidecar. The scoped proof must still establish that no tracked path uses a filter. Exercise all subsets with both hash algorithms, the three-setting diff invocation, and active-filter write and priming attempts with either form.
The reset used by worktree add has no index proof to repair, and its explicit GIT_WORK_TREE prevents proof authentication. A later optional status can establish history, but GIT_OPTIONAL_LOCKS=0 suppresses that priming step. Subsequent read-only status calls cannot persist the missing proof and keep repeating the tracked-file scan. When FSMonitor and writable untracked caching are enabled, let the checkout discover the already registered worktree. Preserve the explicit repository environment for unsupported contexts, including relative configuration-file overrides whose meaning would change with cwd. For a hard reset that creates the index, certify the checkout under the mandatory index lock before committing it. Attach the current config to the new index state, query the provider after checkout, and reuse the writer repair machinery to bind the tracked and untracked proofs. Allow certification under this owned lock even when optional locks are disabled; ordinary read-only status still cannot write or repair its index. Require a complete untracked scan and a closing provider query. Active filters, provider errors, and provider resets must leave the checkout usable without certifying it. Cover those failures, post-checkout edits, and repeated read-only status calls that reuse the proof without writing the index or scanning tracked files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current candidate
Head:
5f1339f46d. This pull request remains review-only; do not merge orenable auto-merge. Fresh approval is required for this head before
controller admission.
Two additional patches address command patterns that prevented proof reuse:
filter.<driver>.cleanandprocesswithrequired=falseas complete read-side disabling, without requiring asmudge override. Scoped FSMonitor reuse still requires a filter-free scope.
including when
GIT_OPTIONAL_LOCKS=0. Later read-only status calls reusethe paired proof without writing the index or scanning tracked files.
Regression controls cover active filters, provider errors and resets,
post-checkout edits, partial overrides, and both hash algorithms.
Focused native validation passed:
t7519-status-fsmonitor.sh: 123 testst7527-builtin-fsmonitor.sh: 200 testst7530-status-clean-sidecar.sh: 67 testsHosted push and pull-request CI each completed with 45 successful jobs and
two expected skips. Style and whitespace checks passed.
A fresh clean developer-mode full suite passed: 33,507 successes, zero
unexpected failures, and 378 known breakages. Full status and FSMonitor
suites also passed at both commit boundaries, as did a separate incremental
build through both patches.
Two executable Git workflows passed on each of macOS and Linux, covering
staging, stash, conflicting rebase, linked worktrees, merge, and worktree
removal. Optimized results matched conservative oracles, and the required
read-only checks left the index unchanged. The macOS workflows used the
same sealed artifact as the latency campaign.
Qualification remains open. An earlier incremental full-suite run failed
the linked-worktree stash-history test in
t7519. Its failing assertionwas not captured. Subsequent clean, incremental, and paired stress runs
pass; branch traces do not support the proposed cause in these patches.
Three further full
t7519repeats per commit also passed. The simulatedreview identified no supported source fix; the original failure remains
recorded and unexplained.
Same-base macOS latency qualification is still open. The corrected
four-repository run passed its oracle, provider-fence, identity, and
read-only-index checks, but did not meet the mean-latency thresholds for
clean writable and dirty read-only status. Median and p95 limits passed.
No samples were discarded and no threshold was relaxed.
A bounded diagnostic captured multi-second stalls in both builds before
Git initialized its process clock; the exact startup cause is unresolved.
These observations are not a latency pass. No release will be admitted
before the remaining latency qualification and fresh approval.
Earlier implementation and validation notes
Summary
open(2)shutdown wake reliable across
EINTR, and containSIGPIPEduring gentleclient writes
accepting a query boundary, with bounded timeout, daemon retirement, token
generation checks, and request coalescing
Controller scope
This review-only pull request presents one prerequisite and two topic patches
on top of
1293167e46. Approve it for controller admission, but do not mergeit into
codex-unstable.Performance
Overlapping Darwin queries whose cookies are already registered share one
provider fence. A timeout or shutdown intersection falls back conservatively
and retires the daemon. Paired same-base latency qualification is being run on
this exact source and binary; this description does not make a latency claim
before that evidence completes.
Validation
At commit
7bed8a334f:t0052-simple-ipc.sh(11/11)unique paths)
EINTRshutdown wake stress (50/50, with 50 verified injections)The preceding candidate,
08f907411e, completed unit tests (406/406),t7519-status-fsmonitor.sh(110/110),t7527-builtin-fsmonitor.sh(194/194),t7530-status-clean-sidecar.sh(59/59), and theASan+UBSan
clean_status_manifestunit tests (4/4). The only tree change fromthat candidate is replacing an open-coded allocation in
t/helper/test-simple-ipc.cwithCALLOC_ARRAY()as required by staticanalysis. Production sources, FSMonitor tests, and the FSMonitor topic
patch-id are unchanged.
Exact-head hosted CI, same-base latency qualification, and executable workflow
qualification are in progress.